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m ■ Abstract 
> ■ 
00 ' 

ly-^ I We perform a comprehensive study of the Higgs sector in the framework of the 

■ next-to-minimal supersymmetric standard model with CP-violating parameters in 
• ' the superpotential and in the soft-supersymmetry- breaking sector. Since the CP is 

■ no longer a good symmetry, the two CP-odd and the three CP-even Higgs bosons of 
. the next-to-minimal supersymmetric standard model in the CP-conserving limit will 

. . i mix. We show explicitly how the mass spectrum and couplings to gauge bosons of the 

.1^ I various Higgs bosons change when the CP-violating phases take on nonzero values. 

^ ' We include full one-loop and the logarithmically enhanced two-loop effects employing 

' the renormalization-group (RG) improved approach. In addition, the LEP limits, the 

global minimum condition, and the positivity of the square of the Higgs-boson mass 
have been imposed. We demonstrate the effects on the Higgs-mass spectrum and 
the couplings to gauge bosons with and without the RG-improved corrections. Sub- 
stantial modifications to the allowed parameter space happen because of the changes 
to the Higgs-boson spectrum and their couplings with the RG-improved corrections. 
Finally, we calculate the mass spectrum and couplings of the few selected scenarios 
and compare to the previous results in literature where possible; in particular, we 
illustrate a scenario motivated by electroweak baryogenesis. 
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1 Introduction 



Supersymmetry (SUSY) is the leading candidate for the physics beyond the standard model 
(SM). It not only solves the gauge hierarchy problem, but also provides a dynamical mecha- 
nism for electroweak symmetry breaking and a natural candidate for the dark matter. The 
minimal supersymmetric extension of the SM (MSSM) has attracted much phenomenolog- 
ical and theoretical interest but it suffers from the so-called little hierarchy problem and 
the fi problem. 

An extension with an extra singlet superfield, known as the next-to-minimal super- 
symmetric standard model (NMSSM) [IHS] was motivated to provide a natural solution to 
the n problem. The fi parameter in the term fiHuHd of the superpotential of the MSSM 
naturally has its value at either Mpianck or zero (due to a symmetry). However, the radia- 
tive electroweak symmetry breaking conditions require the fi parameter to be of the same 
order as the Z-boson mass for fine-tuning reasons. Such a conflict was coined as the /i 
problem [6]. In the NMSSM, the /i term is generated dynamically through the vacuum- 
expectation value (VEV), vs, of the scalar component of the additional Higgs field S, which 
is naturally of the order of the SUSY-breaking scale. Thus, an effective /i parameter of 
the order of the electroweak scale is generated. The NMSSM was recently revived because 
it was shown that it can effectively relieve the little hierarchy problem [7]. Because of 
the additional Higgs singlet field and an approximate Peccei-Quinn (PQ) symmetry, the 
NMSSM naturally has a hght pseudoscalar Higgs boson Oi. It has been shown that, 
in most parameter space that is natural, the SM-like Higgs boson can decay into a pair 
of light pseudoscalar bosons with a branching ratio larger than 0.7. Thus, the branching 
ratio of the SM-like Higgs boson into bb would be less than 0.3 and so the LEPII bound 
is effectively reduced to around 100 GeV [8]. Since the major decay modes of the Higgs 
boson are no longer bb, unusual search modes have been investigated [9]. 

In SUSY models, CP-violating phases naturally appear in the fi term of the su- 
perpotential and in the soft-SUSY breaking terms. The nonobservation of electric dipole 
moments (EDMs) for thallium [10], neutron [H], and mercury p^[T3] is known to constrain 
CP-violating phases very tightly. Nevertheless, cancellations among various contributions 
may occur among several contributions to the three measured EDMs, thus still allowing siz- 
able CP phases [I^[T5] . In the MSSM, the nonvanishing CP phases could radiatively induce 
significant mixing between the CP-even and CP-odd states [T6HT9], giving rise to a number 
of interesting CP-violating phenomena and substantial modifications to Higgs-boson phe- 
nomenology |2ni[2T]. In particular, the lightest Higgs boson can be as light as a few GeV 
with almost vanishing couplings to the weak gauge bosons when the CP-violating phases 
are maximal. The decay patterns of the heavier Higgs bosons become much more compli- 
cated compared to the CP-conserving case because of the loss of its CP parities [221123] • 
These combined features make the Higgs boson-searches at LEP difficult; consequently, the 
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Higgs boson lighter than ~ 50 GeV can survive the LEP hmit [21]. 

In this work, we study the NMSSM Higgs sector with CP violation. In the CP- 
conserving limit, the neutral Higgs sector in the NMSSM includes two CP-odd and three 
CP-even states. With CP violation the 5 neutral Higgs bosons lose their CP parities and 
all mix together. We anticipate a whole new set of phenomena associated with the singlet 
extension of the MSSM in the presence of nontrivial CP-violating phases in the VEVs, and 
the /i and soft SUSY-breaking parameters. As the first step toward this new extension, we 
calculate the whole mass spectrum of the Higgs sector as well as the couplings to the vector 
gauge bosons, which will dictate the production and decay patterns of the Higgs bosons. 
Phenomenology associated with the CP-violating NMSSM Higgs sector will be performed 
in future works. We list a few possible directions at the end of the Conclusions. 

We include in this calculation the important corrections to the Higgs spectrum in 
order to have more precise comparisons to current experimental limits: (i) full one-loop 
corrections to the Higgs-boson masses, and (ii) logarithmically enhanced two-loop correc- 
tions of order 0{g1h'^) and 0{h^) with the renormalization-group (RG) improvement and 
minimization of the two-loop corrections. We also impose highly desirable conditions to 
limit the parameter space: (i) the LEP limits on the Higgs-boson mass and the chargino 
mass limit, (ii) the global minimum condition — the local minimum that we obtain is indeed 
the global minimum, and (iii) the positivity of the square of the Higgs-boson masses. We 
found that the RG-improved corrections have significant reduction in the allowed parame- 
ter space with respect to the LEP limits, the global minimum condition, and the positivity 
of the Higgs-mass squared. 

Before we close this section let us list the parameters of this study. We have (i) 
the usual soft parameters in the MSSM: sfermion masses, A parameters, soft Higgs-boson 
masses, and tan /9; (ii) the additional parameters arisen in NMSSM: A and k in the super- 
potential, A\ and A^. in the soft-breaking sector, and the VEV vs of the singlet Higgs field; 
(iii) the CP phases of A parameters. A, k, and the VEVs (but not all independent). 

The organization of this paper is as follows. We write down the formalism in details 
in the next section, including minimization conditions, combinations of CP phases, tree- 
level mass matrices, and mixing between scalar and pseudoscalar Higgs bosons. In Sec. 
Ill, we calculate the full one-loop and logarithmically enhanced two-loop corrections using 
the RG-improved approach. Numerical presentation for a number of interesting scenarios 
will be demonstrated in Sec. IV, including comparisons to the previous results in literature 
where possible. We conclude and discuss our results in Sec. V. 
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2 Higgs sector at the tree level 



2.1 Conventions 

To begin with, we first introduce the NMSSM superpotentiah 

Wnmssm = U'^KQHu + D'^KHaQ + E^^KHaL + XSH^H^ + S\ (2.1) 

where S denotes the singlet Higgs superfield, Hu,d are the two SU(2)i doublet Higgs super- 
fields, and Q, L and f/*^, D'-', are the matter doublet and singlet superfields, respectively, 
related to up- and down- type quarks and charged leptons. We note that, especially, the last 
cubic term with a dimensionless coupling n respects an extra discrete Z3 symmetry. The 
Yukawa couplings h^^^^e are 3x3 complex matrices describing the quark and charged-lepton 
masses and mixing. In the expression, for example, the notation HuHd = ^ai3{Hu)°'{H(i)^ is 
implicit. The superpotential leads to the tree-level Higgs potential, which is given by the 
sum 

Vo = Vf + Vd + Koft, (2.2) 

where each term is given by 

Vf = \\\^\S\\H\H, + HlH^) + \\H^Ha + >^S^\\ 
Vd = '^^{HlH,-HiH^r + 'liHlH.^)iHiH^ 

l^oft = mlHlH, + mlHiH^ + ml\S\^+(^\A^SH^Hd-^KA,S'' + h.c)j, (2.3) 

with the gauge-coupling constants gi = e/ cos 9w and g2 = e/ sin6'iy. Note the unusual 
minus(— ) sign for the singlet soft-trilinear term proportional to 

Parametrizing the component fields of the two doublet and one singlet scalar Higgs 
fields and the VEVs as follows. 



72^"' 



S = —{vs + (l)l + ias), (2.4) 



*One can go back to the usual convention by taking i?„ — >• — i?„ or 0'^ — > 0'^ + tt in below. 
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we get the following tadpole conditions for the presumed vacuum: 

1 /dVo\ 2 , d + glf 2 2n d , 1^1% 2 , 2n ^t:>VuVI n 



/dVo\ 2 92 + 9f^ 2 2n d , 1^1% 2 , 2n 1 -r, ^d^l 



1 \ 2 D ^dVu , |A| 

\ dcjfs I vs 2 

hvs + =:Ivi = 



1 / dVo\ 2 D ^'^^^ , l-^l^^ 2 , 2n , I |2 2 n c;^ 

/ ^ - mg-Rx V ^-[v^ + vj + \k\ Vg-TlvdVu- RkVs, (2.5) 



Vu\dad/ Vd\dau/ ^ ^ 2 ^ 
1 



= /.^-X.,.„ + /.., = 0, (2.6) 
^'5 \ oas I Vs 



where 



with 



n = |A||k| cos(0';, - 0'^) , 1= |A||k| sin(0';^ - (f)'^) , 
Rx = ^11^ cos(0';^ + 0aJ , Rk = ^-^^^ cos(0'^ + 0aJ , (2.7) 



= + 5 + and 0', = 0, + 3^. (2.^ 



The other parameters Ix and can be reexpressed in terms of X using the CP-odd tadpole 
conditions as 

sm(0^ + 0Aj = -7^2:^5, 



V2 2 

sm(0^ + (/)Aj = -X . (2.9) 



V2 ^"''^ 2 

Therefore, the only rephasing invariant physical CP phase at the tree level is (p'^^ — cf)'^ and, 
once the absolute values of |A|, |Aa|, and \A^^^ are given, the other two combinations of 
CP phases, 0^ + 'pAx and 0'^ + 0yi„, can be determined up to a twofold ambiguity using the 
two CP-odd tadpole conditions in Eq. ( 12. 6p . We also observe that the soft masses m^, m\, 
and m| can be removed using the three CP-even tadpole conditions in Eq. (12. 5p . 

From the potential, with the parametrization of the scalar Higgs fields as in Eq. (12. 4p . 
the scalar mass terms can be derived and they can be cast into the form 

- £^ass = Mf^^H+H- + i $^ A^W^ ^ (2.10) 
where = 0^ sin (3 + (j)^ cos 13 with tan /3 = Vu/vd and 

(0°,0°,0°,a,a5) (2.11) 
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with a = ttd sin /3 + a„ cos /3, rotating away the zero mass Goldstone states. The tree-level 
charged Higgs-boson mass is given by 



M 



(0)2 



(2i?A + -Rvs 



Vs 



lAI 



The 5x5 symmetric mass matrix for the neutral Higgs boson is given by 



(2.12) 



M 



(0)2 
N 



M 
M 
M 



(0)2 



(0)2 



(0)2 



11 
21 
31 



with 



M 



M 



M 



M 



M 



M 



(0)2 



(0)2 



(0)2 



(0)2 



(0)2 



(0)2 



11 



22 



33 



12 



13 



23 



M 
M 
M 



(0)2 



(0)2 
S 

(0)2 



12 
22 
32 



(0)2 



M 
M 

Ms 
\Xvvs 



(0)2 



(0)2 



13 
23 
33 






llvvs 
{Rx - nvs)v 



-^TvuVs 



\ 



2 

2XvdVu 
{Rx - nvs)v 
^-f^ + 2nvdVu + 3R^vs I 



(?.13) 



gl + gl 2 

-v~. 



R 



4 

VdVu 



Ry 



Rx 



+ 2|/tPfo 



-nvs 
\nvs 

RkVs, 



VuVs 

Vd 
VdVs 



M 



M 



M 



(0)2 



(0)2 



(0)2 



21 



31 



32 



9l 



9{ 



4 



+ |A| 



VdVu 



Rx 



-TZvs Vs 



-RxVu + \M^VdVs - TZVuVs, 

-RxVd + \M^VuVs - TlvdVs- 



(2.14) 



2.2 Mixing and mass spectrum with X 7^ 

When I 7^ 0, we should consider the full 5x5 matrix ( I2.13P for the neutral Higgs-boson 
masses and mixing. In this case, the neutral Higgs bosons do not have to carry any definite 
CP parities and its mixing is described by the orthogonal 5x5 matrix Oai as 

(0°,0°,0°,a,a5)'^ = 0^dHi,H2,Hs,H,,H5f (2.15) 

with i^i(5) the lightest (heaviest) Higgs-mass eigenstate. Because of its CP- violating mixing, 
the couplings of the neutral Higgs bosons to the SM and SUSY particles are significantly 
modified. Among them the most eminent one is the couplings of the Higgs bosons to weak 
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gauge bosons in the interaction Lagrangian: 



HHZ 



^-Z,zA Y,9u^yvH^. (2.16) 

(2.17) 



HH±W^ 



-f E ^H....- W^-'^(^.^a, H^) + h.c. , (2.18) 



where the couphngs g„^yv, dn^HjZ 9„ h+w- given in terms of the neutral Higgs-boson 
mixing matrix O by 



9HiVV 
9 HiH+W- 

leading to the following sum rules: 



[{On {cp 02j - Sfs Oij) -{i^ j)] 
C/3 - sp On - iO^i , 



1=1 

5 



1>J 



2,1 |2 

q + 9 



1 — Ogj — for each i . 



(2.19) 



(2.20) 



When the heavier two states have mass larger than that of the other three states, 
they effectively decouple from the mixing and the lighter three states tend to mix among 
themselves. To have a better understanding in this case we introduce the following basis: 



with 



U 



In the $' basis, the symmetric 5x5 mass matrix takes the form 







C/3 








o\ 













1 




























1 








v 














1/ 



M 



(0)2 
N 



(2.21) 



(2.22) 



(2.23) 
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where 



with 







M\ 



Mi 



Vs 



{2Rx + vsn). 



^ sin 2/3 

Then, the tree-level charged Higgs-boson mass can be rewritten as 



lAI 



-V 



(2.24) 



(2.25) 



(2.26) 



On the other hand, the 2x3 heavy-light mixing matrix is 



Ml 



-^^Mls^p-^-fnC2p -\VVSXC2P 



\vvsX 



2vs 



and the symmetric 3x3 mass matrix for the lighter states is given by 



Ml 



2/3 



X 



2 _ A/1 2 
2^ ^2/3 



T ■52/3 



(2.27) 



,2 (Ml „2 



Av% ^2/3 - T*2/3 



2kP- 



1)S 



v 



■|w5Xs2/3 



4^ + — /<■ ■§2/3 



+3 -Rk W5 



(2.28) 



We observe that, in the leading order, A^j ~ M% ~ e Ml, and Ml ~ Ml with 



e = max i S2/3 



In this case, the mass matrix {^M^^^^j could be systematically block diagonalized order 
by order in e, as described in Appendix A. In the block- diagonalized basis, the symmetric 
heavier-state mass matrix becomes 



Ma' Ma' Ma 



(2.29) 



+ Ml ■ 0{e^) 



leading to two almost degenerate eigenmasses 



Ml(^ + ^ sl^ - vh2p (nil ± ^JWrr^^ (2.30) 



and the mass splitting is of order 0{e^). On the other hand, by defining 



(2.31) 



one may obtain the following symmetric mass matrix for the lighter states 



Ml 



which gives rise to the eigenmasses 



V 



M 



Ml + vl 



vs 



±1 



Ml 



Ml 
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2 






+ 


Vs J 







1 2 



Y + vvsS2fjTZ 



3R^vs + -V S2pTl . 



(2.32) 



(2.33) 



We note that the CP-mixing entries in the heavier-state matrix j and the lighter matrix 
M.\ are proportional to the factors v"^! S2/3 or vvs^ S2/3, respectively, and would not affect 
the approximated mass spectrum up to the order O(e^). This could be easily understood 
by observing the CP-mixing entries in A^j^ fl2.27p and (12.281) . which are proportional 
to X, are suppressed by the factor and e^, respectively. In the CP-conserving limit or up 
to the order O(e^), our results agree with those in Ref. |26j . 

When the U{1) PQ symmetry is not broken ot \k,\ = TZ = X = R^/vs = 0, there is no 
CP-violating mixing and the determinant of the lower-right 2x2 submatrix of Eq. (12.131) 
for the CP-odd states vanishes, resulting in a massless CP-odd PQ axion; or else its mass 
is approximately given by Eq. ( 12.33^ . In the same PQ-symmetric limit, the lighter CP- 
even state becomes tachyonic unless F = as seen from Eq. (12.321) . When the U{1) PQ 
symmetry is broken, it is interesting to note that the condition > gives 



2vsMz / I I, 2 p 3 



(2.34) 



M>^^-2|A|X + ^5^2/37^ 
This, together with Eq. (I2.25p . leads to the following constraints on the parameter space 
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in the leading order: 



0<R^ = cos(0', + (PaJ <2vs\^\'. 

M^± ~ Mj^^j^ ~ Ma ~ 



S2/3 

\A,\ ^'^'^^ ^ , (2.35) 

S2/3 COS(0';^ + 0aJ COs(0';^ + 0aJ 

which more or less lift up the twofold ambiguity in using the two CP-odd tadpole conditions 
by fixing 

sign [cos(0'^ + 0aJ] = sign [cos(0';^ + 0a J] = +1 • (2.36) 



2.3 Vacuum condition 

Considering neutral fields only, the effective potential at tree level takes the form 



n'^ ^ n'^ IM2 
,y2"ryi/~2 ~2\2 I \^\ /~2~2 , ~2~2 , ~2~2\ 

+ ^^5 - ]^T^VdVuvl , (2.37) 

where v = ({;„, Vd, Vs, 0, (p) denotes the arbitrary constant fields. The CP phases of the 
VEVs enter into the potential through TZ, Rx, and R^ under the CP-odd tadpole conditions. 
In the NMSSM, diverse vacua can exist. Following the classification discussed in Ref. [25] . 
we define the following phases: 

EW : w ^ 0, VsT^O, 
l:v = 0,vs^O, 
ll:v^0,vs = 0, 
SYM ■.v = vs = 0, (2.38) 



where v = \/v'j + v^. Unlike the MSSM at the tree level, the electroweak-broken vacuum 
(denoted by EW) is not necessarily the global minimum due to the presence of the cubic 
terms in the Higgs potential which are proportional to Rx and R^- To ensure the presumed 
vacuum is the true global minimum of the potential, we require the vacuum energy for the 
chosen vacuum to be smaller than that for any other choices 



I^qI^w ^ Voiv = v)< Voiv ^ v). (2.39) 
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After removing the soft masses mf 2 5 using the CP-even tadpole conditions, the energy 
level of the EW vacuum can be rewritten as 



T/EW 
•^0 



1 



'2/3 



2vsRx + vln 



lAlV 



2 \X\^vl - Ml c 



2 „2 



Z "-2/3 



^ 2 2 

V ^ S2/3 



6 



4 , (2-40) 

where C/3 = cos/3 = v^/v and S/3 = sin/5 = ftj/w. Here, as an example, we compare the 
vacuum energy of the phase EW with that of the phase II. The energy difference, which 
should be positive definite if the EW vacuum is the global minimum, is given by 



T/II _ T/EW 

z z 



gl + gl 

32 



l-l-l 2 2\ 



2 2 



2 2 

vZvi 



\K\ 



(2.41) 



where Vq" denotes the vacuum energy of the phase II with VEVs v = {v^ ,Vu ,0 ,0 ,0). 
Since the two vacua v and v are the simultaneous solutions to the same tadpole conditions, 
the VEVs Vd and Vu are determined by the tadpole conditions taking vs = and using the 
same soft masses ^ g fixed by the EW vacuum. It should be noted that AVq^~^^ can be 
negative if Rx and/or i?^ with the positive signs become large, leading to a metastable EW 
vacuum. Comparison of the EW vacuum with the other vacua also gives similar expressions 
as Eq. ( 12.4ip . Therefore, the requirement for the global minimum can constrain the size of 
\Ax\ and 



3 Higgs sector at the one-loop level 



The one-loop contributions to the Higgs-boson masses can be computed from the effective 
potential 



Str 



log 



(3.1] 



where Qo is the renormalization scale and Ai is the field-dependent mass matrix of all 
modes that couple to the Higgs bosons. The supertrace is defined as Str[/(A^^)] = 
Cj(— l)^**(2sj + l)[/(m^)], where Cj is the color degrees of freedom and Si is the spin of 
the i^^ particle. The field-dependent third-generation quark masses are given by 



m. 



(3.2) 



^ Although a sufficiently long-lived metastable vacuum may be viable, we will not consider such a case 
in this work. 
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where if^^ are the neutral components of Hd^u- The corresponding eigenvalues of the 
squark mass matrices are 



m 



m 



bl,2 



1 

2 

± 

1 
2 

± 



Q ^ U ^ 



Ml - Mi + XtilH'J' - m 



A\ht\'^\AtH^- X*S*{H^ 



o\*\ 



where 



U 2 5 2 



and 



Xb 



9l 



, (3.3) 



(3.4) 



In Eq. (13. 3p . M~, M~, and M~ are real soft-SUSY breaking parameters and At and A^ 
are complex soft-SUSY breaking parameters, Aq = \Ag \ e^'^^i with q = t,b. For the explicit 
expressions of the corrections to the Higgs-boson mass matrix, we refer to Ref. [29] @. 



While the relation between and X remains the same as in Eq. (12.91) . the tree- level 
relation between Ix and X is modified in the presence of the one-loop corrections as 



|A||A; 



sm 



2 ^ 



A/, 



where 



with 



and 



A/, 



q=t,b 

lAI lA 



%/2 



— sin((/)^ 



/(a, 6) 



« 1 log gl - 1 



The one-loop corrected charged Higgs-boson mass is given by 



AM^± 



(3.5) 
(3.6) 

(3.7) 
(3.8) 

(3.9) 



where we refer to Ref. [30] for the explicit form of the correction AM^± . 

In this work, we have included logarithmically enhanced two-loop corrections of the 
order Oi^glh^) and 0{h^), where Qs is the strong gauge coupling. We have adopted the 



addition to the dominant contribution from third- generation quarks and squarks. we have included 
contributions from the weak gauge bosons in our numerical analysis. 
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algorithm suggested in Ref. [31], which incorporates the effects of the RG improvement 
and minimizes the two-loop corrections. Including the full one-loop radiative corrections, 
the 5x5 mass matrix of the neutral Higgs boson can be denoted by 

M% = M^^^' + AMl (3.10) 

where the one-loop correction part may further be decomposed into 

AMl={AM%)^^+{AM%)^^^.^^. (3.11) 

The first term contains the genuine logarithmic contributions which are present even when 
the left-right mixing of the third- generation squarks is absent. The second term describes 
the threshold effects arising from the mass splittings due to the left-right mixing in the 
third-generation squark sectors. The RG-improved mass matrix can be well approximated 
by replacing rrit and in each term by the running masses at appropriate scales as in 



{AMl)^^ ^ (AA^^)ll + (AA^^)mix 

= (AA^^)LLK(/^*)''^^(/^f')] + (A-^^)^iJ"^t(/^?),m,(/i^)] (3.12) 

where the intermediate scales //^ = /ij and /i^ = nit l^i with 

/i? = max (M| + ml, + ml) , 

/i? = max (Mj + m^, M| + m^) . (3.13) 

To obtain the quark masses at the SUSY and intermediate scales, we used the RG equations 
(RGEs) of the two-Higgs doublet model 



- ^ [Qhl + 2hl-W^ml, (3.14) 



dln/i2 647r2 

when 11 > Ma assuming Ma < max(/i^, /i^). When < M^, the SM RGEs are used: 



^ = ^[6(/.rr-6(/.-)^-32,a^L 

^ = ^ [6(/^fT-6(/.n^-32,,2]m?. (3.15) 



4 Numerical analysis 

In this section, we present the numerical results of the RG-improved calculation of the 
masses and mixing matrix of the NMSSM Higgs bosons in the presence of nontrivial CP 
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phases. We make comparisons, where possible, with other RG-improved calculations with- 
out CP phases [32] and the one-loop calculations including CP phases but without the RG 
improvement [29]. The input parameters for the Higgs sector in our numerical study are 
specified as follows: 



tree level 
1-loop level 
CP phases 
signs of 



|A| , \k\ ,tan/3; \Ax\ , \A^\ ,vs 
MQ,M^,Mj^,\At\,\Ak\ 



1/ < J. \ „„„/'^/ 



cos{(l)\ + (/"aJ , cos((/.; + 0aJ . (4.1) 

For the renormalization scale Qq we take the top-quark mass as in Refs. [T7l[T9| [33] . 

On the scenarios under consideration in this section, we have imposed the following 
three conditions: 

• the LEP hmits [21], 

• the global minimum condition requiring that the electroweak vacuum chosen by fix- 
ing the values of tan/3 and vs is the global minimum of the RG-improved effective 
potential, and 

• the positiveness of the Higgs masses squared, M]j- > 0, abandoning the parameter 
space in which one or more of the Higgs states become tachyonic. 



In order to determine whether the prescribed EW vacuum is the global minimum 
or not, we minimize the effective potential numerically. First, the tadpole conditions are 
solved for input VEVs Vi-a and then the solution is fed into the potential. Then the potential 
is numerically minimized in the range {vd ,Vu\c0 \ ,Vu\s8\ ,vs\c^\ ,^51^,^1) < 10 TeV using the 
downhill simplex (Nelder-Mead) method [31]. The output minimum is denoted by Vout- In 
the acceptable cases, the numerically obtained Vont is exactly the same as the input -Uin 
up to the Z3 symmetry. Depending on the parameter space, however, some Vout niay exist 
such that Vefr(vout) < K;ff(i'in)- In most (7^ Vin) corresponds to the VEVs in 

one of the three phases I, II, or SYM (12.381) in which one or both v and vs are vanishing. 
When Vout has Vu,d 7^ and 7^ but with t>out 7^ 246 GeV, it can be made acceptable 
in principle by rescaling the Higgs VEVs to make fout = 246 GeV. But the rescaling may 
require compensating changes of the original values of tan /3 and vs to satisfy the tadpole 
conditions. In this work, we discard such cases by keeping tan /3 and vs fixed. 

Before we go into various scenarios we offer a comment about the range of tan /3. 
It has been shown in Ref. [22] and we have also verified that large tan /3 > 20 GeV is 
not favored by the LEP and global minimum constraints. Furthermore, the CP-violating 
scalar-pseudoscalar mixing is suppressed by large tan/9. We therefore employ a small to 
moderate tan (3 in the following scenarios. 
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4.1 A typical scenario 

We first consider a typical scenario in which the heavier Higgs bosons are relatively light 
by choosing a small tan /3 and a moderate vs [26] : 

tan/3 = 3, vs = 750 GeV , 

Mq = Mjj = M^ = \At\ = \Ab\ = 1000 GeV , 

<P'x = 0, (pAt = <pA, = 0, 

sign [cos(0; + 0^J] = sign [cos(0';, + 0aJ] = +1 , (4.2) 

while varying 

|A|,|fi:|; \Ax\,\An\; 0',. (4.3) 

For definiteness we have fixed Mi = M2 = —200 GeV. 

In Fig. [H we show the allowed region satisfying the three conditions in the |k|-|A| 
plane when \A\\ = 500 GeV and \Ai^\ = 100 GeV for three values of the tree-level CP 
phase: 0'^ = 0° (upper panel), 90° (lower left panel), and 180° (lower right panel). 

The magenta, blue, and red lines denote the LEP limits, the global minimum condi- 
tion, and the positiveness of the Higgs mass squared, respectively. The solid (dotted) lines 
are after (before) the inclusion of the RG improvement. The vertical solid line denotes 
the chargino mass limit, m^± > 104 GeV [35j. For 0'^ = 0°, we observe that the LEP 
constraint is significantly strengthened by the RG improvement, allowing only relatively 
large couplings with |A| > 0.45 and |k| > 0.22. In this regard, the parameter space point 
of |A| = 0.3 and \k\ = 0.1 considered in Ref. [26], which is marginally compatible with the 
LEP limits without the RG improvement, is completely ruled out with the inclusion of the 
RG improvement. For the condition of > 0, we see that too large |A| is not allowed 
when \k,\ < 0.5. This can be understood from Eq. fl2.32p by observing that due to the third 
negative term in the first line and the second term inside the square root below, the lighter 
state mass Ml could become negative when |A| is too large, unless the term 2|Kpw| in the 
first line is large enough. Nevertheless, after imposing the current LEP bounds, we see that 
the condition does not rule out more regions in the |k|-|A| plane. On the other hand, we see 
that the global minimum condition is always stronger than the Mfj > condition and it is 
more restrictive than the LEP bounds in the region 0.55 < IkI < 0.7 when lAI > 0.9. For 
(p'l^ = 90°, the theoretically allowed region is significantly reduced and no region remains 
after applying the LEP limits. For 0'^ = 180°, we still see no region compatible with the 
LEP bounds after the inclusion of the RG improvement. 

In Fig. [21 we show the allowed region in the plane for the three values of 

0'^ = 0° ,90° , and 180°. We have taken |A| = 0.3 and \k\ = 0.1 for comparisons with the 
results presented in [2B], though this point has been ruled out by the LEP limits taking 
into account the RG improvement. Note that we only show the LEP limits without the 
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Figure 1: The allowed region in the |k|-|A| plane for the scenario in Eq. (14 ■2p with l^^l = 500 
GeV and l^^l = 100 GeV for three values of (p'^ = 0° (upper panel), cj)'^ = 90° (lower left 
panel), and 0'^ = 180° (lower right panel). RGI stands for the RG improvement. 
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Figure 2: The allowed region in the IA^I-IAaI plane for the three values of (p'^ = 0° (left 
panel), 90° (middle panel), and 180° (right panel). The scenario in Eq. (14.21) is taken with 
|A| = 0.3 and \k,\ = 0.1. The lines are the same as in Fig. [H 

RG improvement here. By changing 0'^ from 0° to 180°, we observe that the allowed region 
moves to the direction of increasing \Ax\ and \Ai^\. Specifically, |Ak,a| = GeV is not 
allowed when 0'^ > 0° due to the CP-odd tadpole conditions with X 7^ 0; see Eq. (12. 9p . 
We also observe that the allowed region is very small when CP is maximally violated 
with (f)'^ = 90°; see the middle frame. This implies that the inclusion of the CP phases may 
change the phenomenological features of the Higgs sector significantly together with the RG 
improvement. We pursue this issue further in the next section with a scenario compatible 
with the LEP limits. Otherwise, a similar discussion can be applied as in Fig. [D (i) the 
global minimum condition is always stronger than the Mfj > condition and (ii) it further 
constrains the parameter space in addition to the LEP limits. 

The left plot in Fig. [3] shows the numerically obtained VEVs (fout) as a function of 
\Ai^\ taking (f)'^ = and \Ax\ = 500 GeV with |A| = 0.3 and \k\ = 0.1. In the region, 
< < 141 GeV, the prescribed VEVs vew = Vi,^={vd = 78 GeV, Vu = 234 GeV, 
vs = 750 GeV) agree with ■^out and thus satisfy the global minimum condition. For 
\Ak\ > 141 GeV, however, the global minimum moves to phase II with Vu ={vd = 
GeV, Vu = 50 GeV, vs = GeV). The energy levels of the two vacua are plotted in the 
middle panel. The EW vacuum energy (V"eff(fin)) becomes larger as \Af^\ increases. The 
linear dependence is coined from the term appearing in the potential at tree level, 
see Eq. (12.401) . with no lA^I- dependent terms in the one-loop effective potential. On the 
other hand, the vacuum energy of phase II V^^^ = —{92+ 5'i)'«^u/32-|- (one- loop corrections) 
~ —3 X 10^ GeV^ is independent of \Af^\ as shown in the figure. In the right panel, for 
a given value of \Ak\ the effective potential is plotted along the direction which connects 
the two vacua I'll = {vd = GeV, Vu = 50 GeV, vs = GeV) and vew = {vd = 78 GeV, 
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Figure 3: The output VEVs Vu-, Vd, and vs as functions of \A,^\ (left panel) for the scenario in 
Eq. g^D taking |A| = 0.3, \k\ = 0.1, \Ax\ = 500 GeV, and (f)'^ = 0. The middle panel shows 
the vacuum energies for the input and output vacua as functions of \Af^\, which separates 
from each other when l^^l is larger than 141 GeV. In the right panel, the vacuum energies 
along the direction connecting the II (t = 0) and EW {t = 1) vacua are shown for several 
values of 

Vu = 234 GeV, vs = 750 GeV). The direction is parametrized by t such that 

V = (f EW - ^'ii)^ + ^'11 , (4.4) 

and we are moving from the first minimum {vu) to the second extremum [v^w) as the 
parameter t increases from to 1. We note that the EW vacuum is the global minimum 
when 1^4^! = 130 GeV (or < 141 GeV) and it becomes degenerate with the phase II 
vacuum when — 141 GeV. If \Af^\ increases further, the EW vacuum is destabilized or 
M]j < when \Ai^\ ^ 191 GeV and beyond the point, it turns into the maximum as shown 
by the line with \A^\ = 250 GeV. 

Before moving to the next scenario, we examine the renormalization scale (Qo) de- 
pendence of the Higgs-mass spectrum. In Fig. HI we show the masses of the lower three 
(left panel) and the heavier two (right panel) neutral Higgs states in the range between 
Qo = 100 GeV to 1000 GeV. We consider the CP-conserving case with 0'^ = taking 
|A| = 0.3, \k\ = 0.1, \Ax\ = 500 GeV, and \A^\ = 100 GeV. First of all, we observe that 
the implementation of the RG improvement decreases the Qq dependence of the masses. 
The heavier states show the larger variation in their masses than the lighter ones. Among 
the lighter states the singlet CP-odd state H3 is hardly affected by the loop corrections 
due the small value of \k\ = 0.1 and the Hi state is less affected by the RG improvement 
than H2 but shows stronger dependence on Qo- We find similar behaviors in the results 
obtained by using the NMHDECAY code [32]. In the CP-conserving hmit, the NMHDECAY results 
are in agreement with ours after taking into account the Qo dependence and subleading 
contributions ignored in our calculation. 
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Figure 4: The light (left panel) and heavy (right panel) Higgs boson masses as functions 
of the renormalization scale Qo for the scenario in Eq. f l4.2p taking |A| = 0.3, |k| = 0.1, 
|Aa| = 500 GeV, = 100 GeV, and (p'^ = 0, with (sohd lines) and without (dashed lines) 
RGI. 
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Figure 5: The allowed region in the tan P-vs plane for the three values of (p'^ = 0° (left 
panel), 90° (middle panel), and 180° (right panel). We have taken |A| = 0.3, |k| = 0.15, 
\Ax\ = 1200 GeV, and \A^\ = 130 GeV with = = = \At\ = \Ab\ = 1000 GeV, 
= = = 0, Ml = Ma = -200 GeV, and sign [cos (0^ + (/>aJ] = sign [cos(0';, + 
0aJ] = +1. 

4.2 A LEP-compatible scenario 

The scenario considered in the previous section is not compatible with the LEP limits 
after taking into account the RG improvement. Moreover, we observe that the LEP limits 
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Figure 6: The masses M|f,/|M/^. | (left panel) and couplings Qh-w (I'ight panel) as functions 
of tan /3. The dashed lines are at the tree level and the dotted and solid lines at the one-loop 
level without and with the RGI, respectively. We fix vs = 600 GeV while other parameters 
are the same as in Fig. [51 

become very strong with nontrivial CP phases. In this section, we move to a higher value 
of tan/3 toward a LEP-compatible scenario independently of CP phases. 

In Fig. |5l we show the allowed region in the tan s plane for the three values of 
(f)'^ = 0° (left panel), 90° (middle panel), and 180° (right panel). We observe that tan/3 is 
bounded above by ~ 18 due to the LEP and global minimum constraints and further we 
have tan/3 < 13 by applying the chargino mass limit. We note the upper limit on tan/3 
is almost independent of 0'^. In Fig. [6] we show the masses M^./|Mh.| (left panel) and 
couplings Qh-yv (right panel) for the lighter three states as functions of tan /3 while fixing 
Vs = 600 GeV. The mass of H2 is always above the SM LEP limit while Mh^ is below it. 
For tan /3 > 5 where Mfj^ > 0, we find that H3 ~ as and Hi and H2 are mixtures mostly 
of (f)u and 05 §. We find that the mass difference between the two lightest states becomes 
the smallest when tan /3 ~ 10 and Hi becomes dominated by the singlet component there, 
explaining why the coupling QhiVV almost vanishes; see the right frame. The tan/3 value 
at which the resonance occurs could be inferred from Eq. f l2.32p by requiring F = oB or 
tan/3 ~ a/2|74a|/|A|?;5 ~ 10. As tan/3 grows, Mh^ decreases and the size of the coupling 
QhiVV increases, leading us to the fact that the large value of tan /3 is not compatible with 
the LEP limits. 

§ The heavier neutral states are such as H4 ^ a and He, ^ (p^. 
^ We note A/| ~ u| (2|Kp — Rk/vs) for the parameters chosen. 
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Figure 7: The allowed region in the |k|-|A| plane for the scenario in Eq. (14. 5p with = 
1200 GeV and \A^\ = 130 GeV for the three values of 0^ = 0° ,90°, and 180°. The lines 
are the same as in Fig. [H 

Combining these observations, we have fixed the parameters of our LEP-compatible 
scenario as 



tan /3 = 10 , vs = 600 GeV , 

Mq = Mjj = M^ = \At\ = \Ab\ = 1000 GeV , 

= , (pAt = = , 

sign [cos(0'^ + (PaJ] = sign [cos(0';, + </)aJ] = +1 , (4.5) 

varying, again, 

|A|,|«:|; \Ax\,\A^\; 0',. (4.6) 

We have fixed Mi = M2 = —200 GeV as in the previous case. We observe that the effects of 
the CP phases (pAt and (pA,, are negligible in this scenario and we simply take (pAt = (pAt = 0. 

In Fig. [71 we show the allowed region in the |k|-|A| plane taking I^IaI = 1200 GeV 
and = 130 GeV for the three values of 0'^ = 0° , 90°, and 180°. We again see that the 
LEP limits, the global minimum, and the positive Mfj conditions become stronger with 
the RG improvement. Among them, the LEP limits are most strongly constraining the 
parameter space. Especially, when 0'^ = 90° only a small region with 0.25 < |A| < 0.35 and 
\k,\ < 0.2 is allowed after including the chargino mass limit. In Fig. [8], we show the allowed 
region in the IA^I-IAaI plane taking |A| = 0.3 GeV and \k,\ = 0.15. The allowed range of 
\Ax\ is around ~ 1100 GeV when (p'^ = 0° and it moves to a higher- value region as (p'^ 
increases, similar to the scenario considered in the previous section. The LEP limits and 
the positivity condition of Mfj > constrain the parameter space \Ak\ < 250 GeV, and 
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Figure 8: The allowed region in the IAkI-IAaI plane for the scenario ( I4.5P with |A| = 0.3 
and \k\ = 0.15 for the three values of 0'^ = 0° , 90°, and 180°. The lines are the same as in 

Fig.m 

it is further constrained to l^^l < 200 GeV by the global minimum condition. Additional 
restriction arises for small \Ak\ when 0'^ = 90°, 180°. 

To understand why the LEP limits become stronger with the RG improvement and 
why the allowed region shows interesting features when (f)'^ = 90° (see the middle frame 
of Fig. [8]), we examine the three light Higgs-boson masses and their couplings to a pair of 
vector bosons. In Fig. [9l we show the masses (left panel) and couplings (right panel) as 
functions of \Af^\ taking |^;^| = 1200 GeV. In the left panels, the dotted lines are for the 
masses without the RG improvement. The lightest Higgs-boson mass becomes negative 
when I^kI > 250 GeV, signaling the tachyonic state. For (p'^ = (upper panels), two level 
crossings happen at \Af^\ =50 GeV and ~ 70 GeV. Below \Ak\ < 50 GeV, Hi (blue lines) 
is CP odd and the RG improvement pushes down the mass of H2 (black lines) to make the 
LEP limit stronger. When \Ai^\ is between ~ 50 GeV and ~ 70 GeV, H2 is CP odd and 
Hi is near to the LEP limit with sizable coupling gjj-^yy > 10"^. When \Ai^\ > 70 GeV, 
H^ is CP odd and the RG improvement decreases the mass of H2 by the amount of ~ 10 
GeV but is still above the LEP limit, though very near to it. When 0'^ = 90° (lower 
panels), the three states do not carry definite CP parities and gH2VV largest when 

\Ak\ is below 35 GeV and above 75 GeV, explaining the large correction to (black) 
there. Between \Ak\ =35 GeV and 75 GeV, Qh.^yv is the largest and Mh-^ (red) is affected 
by the RG improvement most significantly. We note that glij^yy is enhanced in the region 
around \AJ^ = 70 GeV. This explains why the region is ruled out by the LEP limits, while 
allowed in the CP-conserving case. 

In Fig. [TOl we show the masses (left panels) and the couplings (right panels) of the 
light three Higgs bosons as functions of \Ax\ taking = 130 GeV. Around |Aa| ~ 1200 
GeV, in both cases with 0'^ = 0° and 90°, gn^yv (black lines) is the largest and the RG 
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Figure 9: The masses MfjJ\MH.\ (left panels) and couplings g'jj.yy (right panels) for 
i = 1,2,3 as functions of \AJ^ taking \A\\ = 1200 GeV for the scenario in Eq. (14. 5p . The 
global minimum condition constrains |Aft| as < lA^I < 185 GeV (0'^ = 0°, upper panels) 
and 7 GeV < < 192 GeV (0^ = 90°, lower panels). 

improvement decreases the mass of H2 and increases the mixing between Hi and H2 states, 
making the LEP limits stronger. We observe that the LEP limits allows only the region 
around ~ 1200 GeV since when moving away from the point, the mass of Hi decreases 
while its coupling is rapidly increasing. 
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Figure 10: The masses M^yiMjyJ (left panels) and couplings gn^vv (right panels) for 
i = 1,2,3 as functions of I^a] taking \Ai^\ = 130 GeV for the scenario in Eq. (14. 5p . The 
global minimum condition constrains I^a] as 400 (305) GeV < I^a] < 1995 (2018) GeV 
(0; = 0°, upper panels) and 426 (328) GeV < |Aa| < 2073 (2092) GeV (0^ = 90°, lower 
panels) with (without) the RG improvement. 
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Figure 11: The allowed region in the |k|-|A| plane for the scenario in Eq. (14 .yp with |Aa| = 
600 GeV and 1^4^! = 125 GeV. The lines are the same as in Fig. [H 

4.3 An electroweak baryogenesis (EWBG)-motivated scenario 

The last scenario we are considering has an intermediate value of tan /3 with small vs'- 

tan/3 = 5, vs = 200 GeV , 

Mq = Mjj = M^ = \At\ = \Ab\ = 1000 GeV , 

sign [cos(0'^ + 0aJ] = sign [cos(0';, + 0aJ] = +1 , (4.7) 

while varying 

|A|,|k|; \Ax\,\A,\; (j)',,(j)A, (4.8) 

where (pA = 4>At = (pA^ denotes the common CP phase of the third-generation trilinear 
terms. If it is not mentioned otherwise, we are taking (pA = 0. We have fixed Mi = 
M2 = —200 GeV as in the previous cases. We find that a first-order phase transition could 
occur in some regions of the parameter space of this scenario [25], which is needed for the 
EWBG [36]. 

In Fig. [HI we show the allowed region in the |k|-|A| plane taking I^IaI = 600 GeV 
and \Ak\ = 125 GeV for the three values of 0'^ = 0° (left panel), 90° (middle panel), and 
180° (right panel). Again, we observe that the allowed region is largely affected by the CP 
phase and the RG improvement. The RG improvement tends to shift the allowed region 
to lower values of |A|. The \n\ is bounded above by ~ 0.12 when 4>L = 90°. In Fig. [121 
the allowed region is shown in the |/1k|-|Aa| plane taking |A| = 0.83 [||1 and = 0.05 for 

II It was shown in Rcf. [55] that a |A| = 0.7 at the weak scale is perfectly safe with perturbativity below 
the grand unified theory (GUT) scale, while we have verified that a |A| = 0.8 at the weak scale still gives 
a value below 47r at the GUT scale. Therefore, the value |A| = 0.83 chosen is not expected to have any 
serious violation of perturbativity up to the GUT scale. 
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Figure 12: The allowed region in the IA^I-IAaI plane for the scenario in Eq. (14 .7^ with 
|A| = 0.83 and \k\ = 0.05. The lines are the same as in Fig. [H 

the three values of 0'^ = 0° (left panel), 90° (middle panel), and 180° (right panel). The 
RG improvement tends to shift the allowed region to higher values of \Ax\ and the global 
minimum condition is stronger when 0'^ = 90° and 180°, leaving a small allowed region 
with 100 GeV < U^l < 200 GeV around Ua| ~ 600 GeV. 

r\j I I r\j I ^ I 

In Fig. [131 we show the masses (left panels) and couplings (right panels) as functions 
of \A\\ taking = 125 GeV. The dotted lines are for the masses and couplings without 
the RG improvement. We see that the RG improvement shifts the masses and couplings 
to the region with larger values of |^a|- 

In Fig. [TU we show the masses (left panels) and couplings (right panels) as functions 
of \A^\ {(j)'^) at (f)'^ = 90° (I^kI = 140 GeV) in the upper (lower) frames. We have fixed 
I^IaI = 625 GeV, because for this value the RG improvement could significantly enlarge 
the allowed region as seen from the middle and right panels of Fig. [121 Before including 
the RG improvement, as bounded by the dotted vertical lines, the allowed region is very 
narrow: 

137 GeV < < 144 GeV when 0'^ = 90° (upper) 
88° < 0; < 93° when \A^\ = 140 GeV (lower). 

While, including the RG improvement, the allowed region is enlarged as: 

124 GeV < \A^\ < 152 GeV when 0^ = 90° (upper) 
82°<0'^<18O° when = 140 GeV (lower). 

This is because the couplings to the weak gauge bosons are on the verge of the LEP-allowed 
region and the size of the couplings are generically reduced by the RG improvement, as 
seen from the right panels of Fig. 12. 
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Figure 13: The masses M^./|M//J (left panels) and couplings gfj^yy (right panels) as 
functions of \Ax\ taking \Ai^\ = 125 GeV for the scenario in Eq. (14. 7p with |A| = 0.83, 
\k\ = 0.05. The global minimum condition constrains \Ax\ as 551 GeV < \Ax\ < 625 GeV 
{(j)'^ = 0° without the RGI), 571 GeV < |Aa| < 634 GeV (0^ = 0° with the RGlJ; 576 GeV 
< \Ax\ < 668 GeV (0^ = 90° without the RGI), 596 GeV < |Aa| < 675 GeV (0^ = 90° 
with the RGI). 

Finally, we study the effect of 0^, the common CP phase of the third-generation 
trilinear terms. Being different from the previous LEP-compatible scenario in Eq. (14. 5p . 
we find that the EWBG-motivated scenario in Eq. (14. 7p is sensitive to (pA- In Fig. [151 we 
show the dependence of the allowed region in the |k|-|A| plane on (pA taking \Ax\ = 600 
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Figure 14: (Upper panels) The masses Mll^ (left panels) and couplings g\i.yy (right panels) 
for i = 1, 2, 3 as functions of \Ai^\ taking |Aa| = 625 GeV and 0'^ = 90° for the scenario in 
Eq. (14.71) with |A| = 0.83, |k| = 0.05. (Lower panels) The same as in the upper panels but 
as functions of 0'^ taking \Ai^\ = 140 GeV. The vertical solid and dotted lines bound the 
allowed region with and without the RGI, respectively. 

GeV and \Ak\ = 125 GeV for the three values of 0'^ = 0° (upper panel) and 90° and 180° 
(lower panels). The RG improvement has been included in all cases. The solid lines are 
for (pA = 90° while the dashed lines are for 0^ = 0° which are the same as the solid lines 
in Fig. [11] with the RGI. The bounding lines from the condition Mfj > are not shown 
here because the condition is always weaker than the LEP limits in this case. We observe 
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that the allowed regions prefer lower |A| values when (pA = 90°. For (p'^ = 0° and 90° there 
is no region in which the global minimum condition and the LEP limits can be satisfied 
simultaneously for both values of 0^. In Fig. [161 "we show the dependence of the Higgs 
masses and couplings on (p^ at (f)'^ = 0°. This figure illustrates the interesting case in which 
the CP-conserving limits with (p^ = 0° and 180° are not compatible with the LEP limits. 
The specific parameter set chosen satisfies the LEP constraints only with nontrivial (pA- 
This is because a parametric cancellation occurs between the two terms contributing to 
the coupling gnivv = OuCp + O21S/3 around (pA = 90°, leading to the suppression of the 
coupling, as shown in the right panel. 

5 Conclusions 

We have performed a comprehensive study on the mass spectrum, mixing, and couplings 
to weak gauge bosons of the Higgs sector within the NMSSM, which is a gauge singlet 
extension of the MSSM with Z3 symmetry to address the /i problem. The CP-violating 
parameters in the superpotential and in the soft SUSY-breaking terms are fully taken into 
account. 

At the tree level, there are three rephasing invariant combinations of the CP phases, 
two of which are fixed by the CP-odd tadpole conditions up to a twofold ambiguity. In 
contrast to the MSSM, there still remains one physical CP phase, which induces the tree- 
level CP-violating mixing among the 5 neutral Higgs states. Apart from the CP phase 
{(p\ — (p'^), the tree-level Higgs sector is completely determined by the additional 7 real 
parameters: (z) magnitudes of the two couplings, |A| and (ii) the three VEVs, f„, v^, 
and vs (or v, tan/3, and vs), and (iii) magnitudes of the two A terms, \Ax\ and \Ak\. 
With the general notion of the tree-level CP-violating mixing in the neutral Higgs sector, 
we derive a perturbative way to block-diagonalize a symmetric (n + m) x (n + m) matrix 
iteratively and present analytic expressions for the leading-order effects for the CP- violating 
mixing when the perturbative expansion of the mass matrix works reasonably. 

We have computed the masses and mixing matrix of the Higgs bosons at one-loop 
level using the effective potential method. We have taken into account the CP phases of the 
stop and sbottom sectors, which enter through the combinations of cp'^ + (pAt and (p';^ + (pA^- 
We also include the logarithmically enhanced two-loop corrections of the order 0{g1h'^) 
and 0{h^) by performing the RG improvement of the one-loop effective potential, which 
has been implemented only in the CP-conserving limit before. Beyond tree level our results 
are in agreement with those in the literature, in the CP-conserving limit [32] and in the 
case of without renormalization-group improvement |29] . 
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In our numerical analyses, we have considered the following three different scenarios: 

51 (Typical) : tan/3 = 3 , 1^5 = 750 GeV 

52 (LEP-compatible) : tan/3 = 10 , vg = 600 GeV 

53 (EWBG-motivated) : tan/3 = 5 , vg = 200 GeV 

We have chosen the phase convention with (f)'^ = and varied |A| , |k| ; , ; 0'^ , = 
^At,Ab ■ -^o^ SUSY-breaking parameters we have fixed Mg = M^j = Mj^ = \At\ = \Ab\ = 
1 TeV and, for other parameters, we refer to Eqs. (14.21) . (14.51) . and (14.71) . In each scenario, 
the following three main conditions are imposed to derive constraints on the parameter 
space: (i) the LEP limits, (ii) the global minimum condition, and {Hi) the positivity of the 
Higgs-boson masses squared. The third condition is always weaker than the other two. The 
global minimum condition does not allow too large values for the trilinear parameters \A\\ 
and \Ak\ because the energy of the presumed EW vacuum is proportional to them. The 
LEP limits constrain the allowed parameter space around l^^l ~ \\\vsti3/^/2, which is also 
the typical size of the heavier Higgs bosons. The relative strength of the global minimum 
condition and the LEP limits depends on the scenarios. 

The renormalization-group improvement included in this study substantially strength- 
ens the LEP limits, thus making it more restrictive. In the typical scenario SI, the allowed 
region of the parameter space strongly depends on the CP phase 0'^ and the RG improve- 
ment. We found that the typical points with small |A| and which are allowed before the 
inclusion of the RG improvement, are completely ruled out by the LEP limits after includ- 
ing the RG improvement. In the LEP-compatible scenario S2, the allowed region of the 
parameter space also strongly depends on the CP phase 0'^ and the RG improvement but 
the dependence on (pA is weak. We observe that the RG-improved correction reduces the 
mass of the SM-like Higgs boson by an amount of a few GeV to about 10 GeV, increases the 
mixing between the lighter states, and shrinks the allowed parameter space significantly. 
When (j)'^ takes on nontrivial values, the lighter states do not carry definite CP parities 
and the shape of the allowed parameter region becomes more complicated compared to the 
CP-conserving case. Last, in the EWBG-motivated scenario S3, we find that the global 
minimum condition restricts the parameter space more tightly than the LEP limits and 
some parameter region, which is not allowed in the CP-conserving case, could be allowed 
by assuming nontrivial values of 0'^ and 0a, enlarging the allowed parameter space. 

We offer a few more comments before closing such as the following: 

1. This is the first time that the next-to- minimal supersymmetric standard model is 
studied allowing CP phases in the fi and soft SUSY-breaking parameters, and includ- 
ing full one- loop corrections with renormalization-group improvement. Substantial 
corrections to the Higgs-boson spectrum, mixing, and couplings to weak gauge bosons 
are realized. Furthermore, nontrivial variations in the mass spectrum, mixing, and 
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couplings appear due to nonzero CP phases. Therefore, we anticipate a whole new 
set of phenomenology associated within this CP-violating NMSSM framework. 



2. It is well known that the experimental measured EDMs place nontrivial constraint on 
the CP phases. With one more physical CP phase added in this NMSSM framework 
the predictions for EDMs are important to constrain the combinations of phases |37j . 
Such CP phases are also important to provide enough CP violation required in the 
electroweak baryogenesis. 

3. A successful supersymmetry model should be able to explain the anomalous magnetic 
moment of the muon, which is widely accepted as a 3(j effect [38]. The CP- violating 
NMSSM considered in this work should also be constrained so as to satisfy the muon 
anomaly. It is a nontrivial extension in this regard because of the presence of many 
new Higgs bosons, which can be very light and with CP violating couplings. 

4. There are a number of low-energy constraints on the lightest CP-odd Higgs boson in 
the CP-conserving NMSSM [5^]. The exercises can be repeated in the presence of 
the new CP phases. 

5. There are a number of important cubic terms in the Higgs potential, which also 
have nontrivial dependence on the Higgs spectrum and CP phases. Specifically, for 
successful baryogenesis, the soft cubic term involving the singlet field XA\SHdHu + 
h.c, which is absent in the MSSM, is vital to enable a first-order phase transition 
when the stops are heavy. Apparently, from the EWBG-motivated scenario studied 
in this work, a first-order phase transition is possible in this framework. We will delay 
this issue to a detailed study in the future |37] 

6. A whole new set of phenomenology has to be explored in the Higgs sector with 5 
neutral Higgs bosons with no definite CP parities, and a pair of charged Higgs bosons. 
As we can see in this work, the couplings to weak gauge bosons vary nontrivially with 
the CP phases. The same can be said for the mass spectrum. It is more complicated 
than the CP-violating MSSM or CP-conserving NMSSM. At this point, we cannot 
forecast how the decay branching ratios and production will be modified. We delay 
this to a further study. 

7. With CP violation there is no explicit CP property for the Higgs bosons. In most 
of the cases, there are three relatively light neutral Higgs bosons and two relatively 
heavy ones. The collider phenomenology is particularly concerned with the three 
lighter ones, in which one of them is the SM-like Higgs boson with a relatively large 
coupling to the gauge bosons than the other two Higgs bosons (but still the strength 
is a fraction of the SM value.) At the Tevatron, the most useful Higgs production 
channel is via the associated production with a. W ot a Z boson. The production 
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rate of the SM-like Higgs boson is smaller than the corresponding SM Higgs boson, 
but it may be possible to produce more than one Higgs boson. At the LHC, on the 
other hand, production is dominated by the gluon fusion. The crucial strategies for 
Higgs-boson search depend on the decay pattern of the Higgs bosons. 

8. There are a number of possible channels that the SM-like Higgs boson can decay into, 
including the dominant bb, t^t~ , the rare ones 77, Z'-f, and the possible new 

ones hihi, hih2, and ^12^2 depending on the mass spectrum. In the CP-conserving 
NMSSM, the SM-like Higgs boson can decay into 2 CP-even lighter Higgs bosons or 2 
CP-odd Higgs bosons, but not a mixture. Now with CP violation, the SM-like Higgs 
boson can decay into hih2, which is not possible in the CP-conserving case. It is a 
clean signal of CP violation. Further decays of hi and /i2 will give a total of four 
fermions in the final state, e.g., 46, 262r, 4r, 2/x2r, 262/i, etc. Feasibility and 
coverage of parameter space certainly deserve further studies. 

We conclude by summarizing that we have started a new avenue in the CP-violating 
NMSSM, which involves a whole new set of phenomenology in low-energy precision mea- 
surements, in the LHC Higgs-boson searches, baryogenesis, etc., to be explored in the 
future. 
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A Appendix 

In this Appendix, we consider a block diagonalization of a symmetric {n + m) x [n + m) 
matrix 

where the submatrices can be expanded as 

A = Ao + 5^eM„ ; 5 = ^e"S„ ; C = ^e"C„. (A.2) 

n=l n=l n=l 

Note that, for successful diagonalization, we require the off-diagonal n x m submatrix C 
and the lower m x m submatrix B to be suppressed by, at least, one power of e compared 
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with the upper diagonal nxn submatrix A. The block diagonalization can be implemented 
by introducing the mixing matrix 

V^f'Zlt" 1 % J . (A.3) 

where the submatrices in the diagonal parts are symmetric, = y and = z, and all 
three submatrices can also be expanded as F*l 

x = J2^''^n ; ; z = J2^''^n. (A.4) 

n=l n=2 n=2 

Using the orthogonality of the matrix V, VV^ = V'^V = l(n+m)x{n+m), we impose the 
relations 

2y + + xx^ = Onxn ; 2z + z^ + x^x = Omxm ] yx = xz , (A. 5) 

which can be used to determine the diagonal matrices y and z in terms of x iteratively, 
order by order in e, as follows: 

1 ^ 
2/2 = -^xix^ 

2/4 = (3:1X3 + xsxf + X2X2) - hxixj^f 
Z o 

yi = yi{xi,x2, ■ ■ ■ ,Xi-i) (A. 6) 



and 



yi (xj — > xj for j < ij (A. 7) 



Therefore, to determine the diagonal entries of the mixing matrix V up to order e*, all 
we need to know is {xi,X2, ■ ■ ■ ,Xi-i) which can be obtained by requiring the off-diagonal 
part of VSV"^ to vanish, order by order in e up to e*~^, and in terms of the submatrices 
^0,1, .■■,(«-!)' -Si,. ..,(«-!)! and Ci_2,...,(i-i)- 

More specifically we define the block-diagonal matrix 



5*11 S_ 



S ^ VSV^' = ( ] , (A.8) 

'-'12 ^22 



**Note that y and z starts from the second order of e, reqmred by the e expansion of the off-diagonal 
part X oi V and the orthogonahty of the mixing matrix V, as shown below. 
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with 

S^^ = A+ {Ay + yA + Cx^ + xC'^) + xBx^ + {yAy + yCx^ + xC'^y) 

= B + {x^Ax-C^x-x'^C) + {Bz + zB) + {-zC^x-x^Cz) + zBz (A.9) 

in order of increasing power in e of the leading terms. The vanishing off-diagonal part is 

= {-Ax + C) + xB + {-yAx-xC^x + yC + Cz)+xBz + yCz , (A.IO) 

where the first, the second, the third, the fourth, and the fifth term starts from e^, e^, e'^, 
e^, and e^, respectively. To solve S12 = Onxm, we have made the following rearrangement: 

5i2 = 5^e^(5i2)^ = ^e*(-AoX, + a + A) , (A.ll) 

i=l * i=l 

where A's are functions of A, B, C, and (xi,X2, . . . ,Xi-i): 

Di = Di{xj] Aj, Bj, Cj) with j <i . (A.12) 
Therefore, (5*12)1 = Onxm can be solved iteratively to give xf 

x, = A^'{a + D,) (A.13) 
Here, for example, we give a few first -Dj's: 
= 

D2 = -A,xi+xiB, (A.14) 

-D3 = -A1X2 - A2X1 + X2B1 + X1B2 - -CiXi xi - -xiXi Ci - xiCi xi + -xiXi AqXi . 

This completes the block diagonalization of the symmetric matrix S. To summarize, assum- 
ing all the Xj^s are known up to j — i — 1, can be easily obtained by solving (>S'i2)i — Onxm 
and then [xi,X2, ■■■,Xi) fixes and Zi_^.i for the mixing matrix and the block-diagonalized 
matrices up to the e*"*"^ order. 

As a simple application of our method, we consider the situation [ill 

A = Ao, B = e^B2, C = eCi. (A.15) 

In the first order of e, 

xi = VCi, (A.16) 

which leads to 11 

y2 = --xixf , Z2 = --xf xi , (A.17) 

for the mixing matrix and the block-diagonalized matrices are given by 

Su = Ao + ^e\Cix^ + xiCf); S22 = e\B2-Clx,). (A.18) 
Note that, in this simple case, X2 and, accordingly, 7/3 and Z3 vanish. 



''"''See, also, the Appendix in Ref. [25] , 
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Figure 15: The allowed region in the |k|-|A| plane for the two values of (j)A = 0° (dashed 
lines) and 90° (solid lines) taking the scenario in Eq. (14 .yp with |Aa| = 600 GeV, l^^l = 125 
GeV and the three values of (f)'^ = 0° (upper panel) and 90° and 180° (lower panels). The 
RG improvement has been included in all cases. The dashed lines for (pA = 0° are the same 
as the solid lines in Fig. [HI 
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Figure 16: The masses Mh. (left panel) and couplings gj^.w (right panel) for z = 1, 2, 3 as 
functions of (pA taking l^^l = 125 GeV and I^a] = 600 GeV for the scenario in Eq. (14.71) 
with |A| = 0.815, \k\ = 0.07, and 0'^ = 0°. The vertical solid and dotted lines bound the 
allowed region with and without the RGI, respectively. 
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